{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "44b9976d",
      "metadata": {},
      "source": [
        "# How to split code\n",
        "\n",
        ":::info Prerequisites\n",
        "\n",
        "This guide assumes familiarity with the following concepts:\n",
        "\n",
        "- [Text splitters](/docs/concepts/text_splitters)\n",
        "- [Recursively splitting text by character](/docs/how_to/recursive_text_splitter)\n",
        "\n",
        ":::\n",
        "\n",
        "\n",
        "[RecursiveCharacterTextSplitter](https://api.js.langchain.com/classes/langchain_textsplitters.RecursiveCharacterTextSplitter.html) includes pre-built lists of separators that are useful for splitting text in a specific programming language.\n",
        "\n",
        "Supported languages include:\n",
        "\n",
        "```\n",
        "\"html\" | \"cpp\" | \"go\" | \"java\" | \"js\" | \"php\" | \"proto\" | \"python\" | \"rst\" | \"ruby\" | \"rust\" | \"scala\" | \"swift\" | \"markdown\" | \"latex\" | \"sol\"\n",
        "```\n",
        "\n",
        "To view the list of separators for a given language, pass one of the values from the list above into the `getSeparatorsForLanguage()` static method"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "c92fb913",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  \u001b[32m\"\\nfunction \"\u001b[39m, \u001b[32m\"\\nconst \"\u001b[39m,\n",
              "  \u001b[32m\"\\nlet \"\u001b[39m,      \u001b[32m\"\\nvar \"\u001b[39m,\n",
              "  \u001b[32m\"\\nclass \"\u001b[39m,    \u001b[32m\"\\nif \"\u001b[39m,\n",
              "  \u001b[32m\"\\nfor \"\u001b[39m,      \u001b[32m\"\\nwhile \"\u001b[39m,\n",
              "  \u001b[32m\"\\nswitch \"\u001b[39m,   \u001b[32m\"\\ncase \"\u001b[39m,\n",
              "  \u001b[32m\"\\ndefault \"\u001b[39m,  \u001b[32m\"\\n\\n\"\u001b[39m,\n",
              "  \u001b[32m\"\\n\"\u001b[39m,          \u001b[32m\" \"\u001b[39m,\n",
              "  \u001b[32m\"\"\u001b[39m\n",
              "]"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import {\n",
        "  RecursiveCharacterTextSplitter,\n",
        "} from \"@langchain/textsplitters\";\n",
        "\n",
        "RecursiveCharacterTextSplitter.getSeparatorsForLanguage(\"js\");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "354f60a5",
      "metadata": {},
      "source": [
        "## JS\n",
        "Here's an example using the JS text splitter:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "7db0d486",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  Document {\n",
              "    pageContent: \u001b[32m'function helloWorld() {\\n  console.log(\"Hello, World!\");\\n}'\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m2\u001b[39m, to: \u001b[33m4\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"// Call the function\\nhelloWorld();\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m6\u001b[39m, to: \u001b[33m7\u001b[39m } } }\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const JS_CODE = `\n",
        "function helloWorld() {\n",
        "  console.log(\"Hello, World!\");\n",
        "}\n",
        "\n",
        "// Call the function\n",
        "helloWorld();\n",
        "`\n",
        "\n",
        "const jsSplitter = RecursiveCharacterTextSplitter.fromLanguage(\n",
        "    \"js\", {\n",
        "      chunkSize: 60,\n",
        "      chunkOverlap: 0,\n",
        "    }\n",
        ")\n",
        "const jsDocs = await jsSplitter.createDocuments([JS_CODE]);\n",
        "\n",
        "jsDocs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dcb8931b",
      "metadata": {},
      "source": [
        "## Python\n",
        "\n",
        "Here's an example for Python:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "a58512b9",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  Document {\n",
              "    pageContent: \u001b[32m'def hello_world():\\n    print(\"Hello, World!\")'\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m2\u001b[39m, to: \u001b[33m3\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"# Call the function\\nhello_world()\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m5\u001b[39m, to: \u001b[33m6\u001b[39m } } }\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const PYTHON_CODE = `\n",
        "def hello_world():\n",
        "    print(\"Hello, World!\")\n",
        "\n",
        "# Call the function\n",
        "hello_world()\n",
        "`\n",
        "\n",
        "const pythonSplitter = RecursiveCharacterTextSplitter.fromLanguage(\n",
        "    \"python\", {\n",
        "        chunkSize: 50,\n",
        "        chunkOverlap: 0,\n",
        "    }\n",
        ")\n",
        "const pythonDocs = await pythonSplitter.createDocuments([PYTHON_CODE])\n",
        "pythonDocs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ee2361f8",
      "metadata": {},
      "source": [
        "## Markdown\n",
        "\n",
        "Here's an example of splitting on markdown separators:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "ac9295d3",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"# 🦜️🔗 LangChain\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m2\u001b[39m, to: \u001b[33m2\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"⚡ Building applications with LLMs through composability ⚡\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m4\u001b[39m, to: \u001b[33m4\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"## Quick Install\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m6\u001b[39m, to: \u001b[33m6\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"```bash\\n# Hopefully this code block isn't split\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m8\u001b[39m, to: \u001b[33m9\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"pip install langchain\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m10\u001b[39m, to: \u001b[33m10\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"```\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m11\u001b[39m, to: \u001b[33m11\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"As an open-source project in a rapidly developing field, we\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m13\u001b[39m, to: \u001b[33m13\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"are extremely open to contributions.\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m13\u001b[39m, to: \u001b[33m13\u001b[39m } } }\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const markdownText = `\n",
        "# 🦜️🔗 LangChain\n",
        "\n",
        "⚡ Building applications with LLMs through composability ⚡\n",
        "\n",
        "## Quick Install\n",
        "\n",
        "\\`\\`\\`bash\n",
        "# Hopefully this code block isn't split\n",
        "pip install langchain\n",
        "\\`\\`\\`\n",
        "\n",
        "As an open-source project in a rapidly developing field, we are extremely open to contributions.\n",
        "`;\n",
        "\n",
        "const mdSplitter = RecursiveCharacterTextSplitter.fromLanguage(\n",
        "  \"markdown\", {\n",
        "      chunkSize: 60,\n",
        "      chunkOverlap: 0,\n",
        "    }\n",
        ")\n",
        "const mdDocs = await mdSplitter.createDocuments([markdownText])\n",
        "\n",
        "mdDocs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7aa306f6",
      "metadata": {},
      "source": [
        "## Latex\n",
        "\n",
        "Here's an example on Latex text:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "77d1049d",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"documentclass{article}\\n\\n\\begin{document}\\n\\nmaketitle\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m2\u001b[39m, to: \u001b[33m6\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"section{Introduction}\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m8\u001b[39m, to: \u001b[33m8\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"Large language models (LLMs) are a type of machine learning\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m9\u001b[39m, to: \u001b[33m9\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"model that can be trained on vast amounts of text data to\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m9\u001b[39m, to: \u001b[33m9\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"generate human-like language. In recent years, LLMs have\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m9\u001b[39m, to: \u001b[33m9\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"made significant advances in a variety of natural language\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m9\u001b[39m, to: \u001b[33m9\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"processing tasks, including language translation, text\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m9\u001b[39m, to: \u001b[33m9\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"generation, and sentiment analysis.\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m9\u001b[39m, to: \u001b[33m9\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"subsection{History of LLMs}\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m11\u001b[39m, to: \u001b[33m11\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"The earliest LLMs were developed in the 1980s and 1990s,\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m12\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"but they were limited by the amount of data that could be\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m12\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"processed and the computational power available at the\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m12\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"time. In the past decade, however, advances in hardware and\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m12\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"software have made it possible to train LLMs on massive\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m12\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"datasets, leading to significant improvements in\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m12\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"performance.\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m12\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"subsection{Applications of LLMs}\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m14\u001b[39m, to: \u001b[33m14\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"LLMs have many applications in industry, including\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m15\u001b[39m, to: \u001b[33m15\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"chatbots, content creation, and virtual assistants. They\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m15\u001b[39m, to: \u001b[33m15\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"can also be used in academia for research in linguistics,\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m15\u001b[39m, to: \u001b[33m15\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"psychology, and computational linguistics.\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m15\u001b[39m, to: \u001b[33m15\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"end{document}\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m17\u001b[39m, to: \u001b[33m17\u001b[39m } } }\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const latexText = `\n",
        "\\documentclass{article}\n",
        "\n",
        "\\begin{document}\n",
        "\n",
        "\\maketitle\n",
        "\n",
        "\\section{Introduction}\n",
        "Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.\n",
        "\n",
        "\\subsection{History of LLMs}\n",
        "The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.\n",
        "\n",
        "\\subsection{Applications of LLMs}\n",
        "LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.\n",
        "\n",
        "\\end{document}\n",
        "`\n",
        "\n",
        "const latexSplitter = RecursiveCharacterTextSplitter.fromLanguage(\n",
        "  \"latex\", {\n",
        "      chunkSize: 60,\n",
        "      chunkOverlap: 0,\n",
        "    }\n",
        ")\n",
        "const latexDocs = await latexSplitter.createDocuments([latexText])\n",
        "\n",
        "latexDocs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c29adadf",
      "metadata": {},
      "source": [
        "## HTML\n",
        "\n",
        "Here's an example using an HTML text splitter:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "0fc78794",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<!DOCTYPE html>\\n<html>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m2\u001b[39m, to: \u001b[33m3\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<head>\\n        <title>🦜️🔗 LangChain</title>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m4\u001b[39m, to: \u001b[33m5\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<style>\\n            body {\\n                font-family:\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m6\u001b[39m, to: \u001b[33m8\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"Arial, sans-serif;\\n            }\\n            h1 {\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m8\u001b[39m, to: \u001b[33m10\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"color: darkblue;\\n            }\\n        </style>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m11\u001b[39m, to: \u001b[33m13\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"</head>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m14\u001b[39m, to: \u001b[33m14\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<body>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m15\u001b[39m, to: \u001b[33m15\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<div>\\n            <h1>🦜️🔗 LangChain</h1>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m16\u001b[39m, to: \u001b[33m17\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<p>⚡ Building applications with LLMs through composability\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m18\u001b[39m, to: \u001b[33m18\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"⚡</p>\\n        </div>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m18\u001b[39m, to: \u001b[33m19\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<div>\\n            As an open-source project in a rapidly\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m20\u001b[39m, to: \u001b[33m21\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"developing field, we are extremely open to contributions.\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m21\u001b[39m, to: \u001b[33m21\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"</div>\\n    </body>\\n</html>\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m22\u001b[39m, to: \u001b[33m24\u001b[39m } } }\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const htmlText = `\n",
        "<!DOCTYPE html>\n",
        "<html>\n",
        "    <head>\n",
        "        <title>🦜️🔗 LangChain</title>\n",
        "        <style>\n",
        "            body {\n",
        "                font-family: Arial, sans-serif;\n",
        "            }\n",
        "            h1 {\n",
        "                color: darkblue;\n",
        "            }\n",
        "        </style>\n",
        "    </head>\n",
        "    <body>\n",
        "        <div>\n",
        "            <h1>🦜️🔗 LangChain</h1>\n",
        "            <p>⚡ Building applications with LLMs through composability ⚡</p>\n",
        "        </div>\n",
        "        <div>\n",
        "            As an open-source project in a rapidly developing field, we are extremely open to contributions.\n",
        "        </div>\n",
        "    </body>\n",
        "</html>\n",
        "`\n",
        "\n",
        "const htmlSplitter = RecursiveCharacterTextSplitter.fromLanguage(\n",
        "  \"html\", {\n",
        "      chunkSize: 60,\n",
        "      chunkOverlap: 0,\n",
        "    }\n",
        ")\n",
        "const htmlDocs = await htmlSplitter.createDocuments([htmlText])\n",
        "htmlDocs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fcaf7abf",
      "metadata": {},
      "source": [
        "## Solidity\n",
        "Here's an example using of splitting on [Solidity](https://soliditylang.org/) code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "49a1df11",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"pragma solidity ^0.8.20;\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m2\u001b[39m, to: \u001b[33m2\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"contract HelloWorld {\\n\"\u001b[39m +\n",
              "      \u001b[32m\"   function add(uint a, uint b) pure public returns(uint) {\\n\"\u001b[39m +\n",
              "      \u001b[32m\"       return a + \"\u001b[39m... 9 more characters,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m3\u001b[39m, to: \u001b[33m7\u001b[39m } } }\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const SOL_CODE = `\n",
        "pragma solidity ^0.8.20;\n",
        "contract HelloWorld {\n",
        "   function add(uint a, uint b) pure public returns(uint) {\n",
        "       return a + b;\n",
        "   }\n",
        "}\n",
        "`\n",
        "\n",
        "const solSplitter = RecursiveCharacterTextSplitter.fromLanguage(\n",
        "    \"sol\", {\n",
        "        chunkSize: 128,\n",
        "        chunkOverlap: 0,\n",
        "    }\n",
        ")\n",
        "const solDocs = await solSplitter.createDocuments([SOL_CODE])\n",
        "solDocs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4a11f7cd-cd85-430c-b307-5b5b5f07f8db",
      "metadata": {},
      "source": [
        "## PHP\n",
        "Here's an example of splitting on PHP code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "90c66e7e-87a5-4a81-bece-7949aabf2369",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"<?php\\nnamespace foo;\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m1\u001b[39m, to: \u001b[33m2\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"class Hello {\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m3\u001b[39m, to: \u001b[33m3\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"public function __construct() { }\\n}\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m4\u001b[39m, to: \u001b[33m5\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m'function hello() {\\n    echo \"Hello World!\";\\n}'\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m6\u001b[39m, to: \u001b[33m8\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"interface Human {\\n    public function breath();\\n}\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m9\u001b[39m, to: \u001b[33m11\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"trait Foo { }\\nenum Color\\n{\\n    case Red;\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m12\u001b[39m, to: \u001b[33m15\u001b[39m } } }\n",
              "  },\n",
              "  Document {\n",
              "    pageContent: \u001b[32m\"case Blue;\\n}\"\u001b[39m,\n",
              "    metadata: { loc: { lines: { from: \u001b[33m16\u001b[39m, to: \u001b[33m17\u001b[39m } } }\n",
              "  }\n",
              "]"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const PHP_CODE = `<?php\n",
        "namespace foo;\n",
        "class Hello {\n",
        "    public function __construct() { }\n",
        "}\n",
        "function hello() {\n",
        "    echo \"Hello World!\";\n",
        "}\n",
        "interface Human {\n",
        "    public function breath();\n",
        "}\n",
        "trait Foo { }\n",
        "enum Color\n",
        "{\n",
        "    case Red;\n",
        "    case Blue;\n",
        "}`\n",
        "\n",
        "const phpSplitter = RecursiveCharacterTextSplitter.fromLanguage(\n",
        "    \"php\", {\n",
        "        chunkSize: 50,\n",
        "        chunkOverlap: 0,\n",
        "    } \n",
        ")\n",
        "const phpDocs = await phpSplitter.createDocuments([PHP_CODE])\n",
        "\n",
        "phpDocs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "22ade4ef",
      "metadata": {},
      "source": [
        "## Next steps\n",
        "\n",
        "You've now learned a method for splitting text on code-specific separators.\n",
        "\n",
        "Next, check out the [full tutorial on retrieval-augmented generation](/docs/tutorials/rag)."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Deno",
      "language": "typescript",
      "name": "deno"
    },
    "language_info": {
      "file_extension": ".ts",
      "mimetype": "text/x.typescript",
      "name": "typescript",
      "nb_converter": "script",
      "pygments_lexer": "typescript",
      "version": "5.3.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}